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I.  INTRODUCTION 


A.  BACKGROUND 

The  current  method  of  choice  for  developing  risk-adjusted  cost  estimates  is  the 
Monte  Carlo  simulation  method,  which  sums  probability-based  cost  distributions  at  the 
work  breakdown  structure  (WBS)  element  level  to  produce  a  cumulative  distribution  for 
overall  program  cost.  The  confidence  level  from  the  cumulative  distribution  is  the 
resulting  “risk”  for  the  estimated  cost.  This  Monte  Carlo  method  is  also  referred  to 
within  the  cost-estimating  community  as  the  s-curve  method  of  cost-risk  analysis,  and 
although  it  is  the  current  method  of  choice  for  developing  risk-adjusted  cost  estimates,  it 
leaves  several  of  the  following  problems  unmanaged: 

1.  Programs  have  not  behaved  as  the  cost  estimation  model  had  forecast.  The 
cost  risk  becomes  a  second-order  guess  based  on  another  guess. 

2.  This  model  does  not  bound  possible  risk  areas  both  on  the  high  and  low 
likelihood  probability.  By  using  this  method,  the  left  and  right  tails  of  the 
cumulative  distribution  are  weighted  equally  with  the  mean. 

3.  The  model  assumes  that  the  consequences  for  all  unfavorable  outcomes 
equally  affect  the  cost  model. 

4.  The  model  requires  in-depth  knowledge  of  statistical  analysis  and  statistical 
processes. 

The  s-curve  method  of  cost-risk  analysis  for  major  Department  of  Defense  (DoD) 
and  Department  of  the  Navy  (DoN)  acquisitions  projects,  so-called  major  defense 
acquisition  programs  (MDAPs),  does  not  accurately  estimate  actual  cost  from  the  pre- 
Milestone  B  stages  of  the  acquisitions  process  to  the  phase  at  which  the  program  reaches 
full-rate  production;  thus,  the  s-curve  method  produces  unreliable  cost  risk  analysis. 

One  of  the  reasons  that  cost  estimates  at  Milestone  B  are  inaccurate  is  that  very 
little,  if  any,  real  information  about  the  project  is  known  other  than  the  conceptual 
capabilities  required. 
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The  enhanced  scenario-based  method  (eSBM)  makes  risk  assessments  without  the 
need  for  details  about  the  program,  while  still  being  grounded  in  historical  facts  and  data. 
eSBM  allows  managers  a  less  statistically  taxing  method  of  determining  cost  risk  for  a 
project. 


1.  Purpose 

The  key  factors  in  measuring  the  usefulness  of  eSBM  should  be  to  focus  on  the 
acquisition  strategy  being  used  for  the  project  and  the  time  frame  from  Milestone  B  to 
later  milestones.  In  this  thesis,  I  hypothesize  that  different  acquisition  strategies  will 
yield  different  levels  of  variation  in  estimating  cost  risk  for  eSBM.  Furthermore,  I 
hypothesize  that  the  longer  a  program  takes  to  reach  full-rate  production  from  the 
Milestone  B,  the  more  cost  risk  there  is  for  that  program. 

The  eSBM  method  is  not  meant  to  replace  use  of  statistical  analysis  to  determine 
cost  risk;  it  is  intended  to  provide  an  alternative  to  the  current  standard.  By  having  an 
additional  method  of  risk  analysis,  cost  estimators  and  program  managers  are  provided 
with  yet  another  risk  profile  with  which  to  compare  and  measure  risk. 

2.  The  Question 

There  is  no  definitive  research  that  links  the  effects  of  acquisition  strategy  and/or 
program  length  to  program  cost  risk  and  the  Monte  Carlo  method  of  cost-risk  analysis 
makes  no  attempt  to  factor  in  acquisition  strategy  or  the  time  horizon  for  the  program. 
On  the  other  hand,  the  eSBM  makes  an  effort  to  reduce  statistical  uncertainty  in 
estimating  costs  and  reducing  the  risk  of  cost  overrun.  However,  it  falls  short  in  the  areas 
of  acquisition  strategy  and  time  horizon.  I  hypothesize  that  different  acquisition 
strategies  will  yield  different  levels  of  risk  in  estimating  cost  risk  for  eSBM. 

3.  Who  Benefits 

More  accurate  risk  analysis  benefits  every  level  of  DoD  decision-making.  The 
following  examples  outline  some  of  these  benefits. 

•  By  providing  an  accurate  assessment  of  a  program’s  cost  risk,  the  senior 

acquisition  managers  and  Milestone  Decision  Authorities  (MDA)  will  be 
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able  to  manage  program  risk  early  on  and  avoid  potentially  overly  risky 
programs.  Having  another  cost-risk  profile  will  allow  for  more  rational 
debates  during  the  analysis  of  alternatives  (AOA)  stage  of  the  acquisition 
process.  By  weeding  out  programs  with  a  high  likelihood  of  failure,  more 
funds  will  be  available  for  other  programs  that  are  likely  to  succeed. 

•  Program  executive  offices  (PEOs)  and  program  managers  (PMs)  will  be 
able  to  focus  attention  on  areas  of  highest  risk  sooner  in  the  program,  thus 
detecting  the  potential  effects  on  cost,  schedule,  and  perfonnance  before 
problems  arise.  The  more  accurate  cost-risk  assessment  would  also  allow 
for  realistic  decisions  about  how  to  manage  the  program  as  a  whole. 

•  Cost  estimators  would  be  provided  a  method  of  analysis  that  is  founded  in 
historical  data  without  the  rigorous  statistical  analysis.  Due  to  the 
complex  statistical  analysis  methods  needed  to  produce  accurate  Monte 
Carlo  simulations,  cost  estimations  are  subject  to  additional  scrutiny  from 
statisticians.  The  eSBM  method  removes  this  statistical  scrutiny  while 
still  maintaining  the  desired  outcome  in  terms  familiar  to  decision-makers. 

In  a  dynamic  world  of  fiscal  constraints,  accurate  assessment  of  cost-estimating 
risk  is  even  more  crucial. 

4.  The  Next  Steps 

In  the  next  chapter,  I  describe  how  the  eSBM  was  developed,  how  its  parameters 
were  derived,  and  how  it  is  used  for  cost-risk  analysis.  In  that  section,  I  include  a 
description  of  the  methods  used  to  construct  the  coefficients  of  variation  (CV)  for  the 
different  cost  drivers.  Additionally,  I  explain  how  to  translate  the  eSBM  results  into 
tenns  familiar  to  the  acquisition  community. 

In  this  thesis,  I  present  my  research  of  how  acquisition  strategy  and  program 
length  affect  the  cost-risk  profile  of  MDAPs.  This  research  consists  of  deriving  the  cost 
growth  factors  (CGF)  and  coefficients  of  variation  by  using  historical  data.  After  the 
CVs  are  formulated,  I  tested  them  for  validity  and  translated  into  a  cost  risk  profile. 
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II.  LITERATURE  REVIEW 


A.  INTRODUCTION 

In  2005,  Paul  Garvey  introduced  the  scenario-based  method  (SBM)  as  an 
alternative  to  the  s-curve  method  of  risk-adjusted  cost  estimation;  he  described  it  as  an 
effective,  less  statistically  intensive  method  of  determining  cost-risk  analysis  (Garvey, 
2005).  The  Air  Force  introduced  the  SBM  in  the  2007  Cost  Risk  and  Uncertainty 
Handbook  as  an  acceptable  method  to  calculate  cost  estimate  uncertainty  (Air  Force  Cost 
Analysis  Agency  [AFCAA],  2007). 

In  order  to  meet  the  requirements  of  the  Weapons  System  Acquisition  Reform  Act 
(WSARA,  2009),  the  SBM  would  need  to  be  able  to  produce  confidence  interval  data  that 
could  be  compared  in  the  same  manner  as  the  s-curve  analysis.  Additionally,  the  new 
method  would  need  to  be  coupled  with  historical  data  from  previous  defense  acquisition 
programs.  These  requirements  have  led  to  the  development  of  the  statistical  SBM,  which 
uses  simple  algebraic  formulas  and  basic  statistical  processes  to  generate  cost  risk  curves. 
Flynn  and  Garvey  conducted  research  on  historical  cost  data  to  produce  historically 
driven  CVs,  lending  historical  credibility  to  the  statistical  SBM.  This  historical  data  led  to 
the  eSBM. 

B.  WEAPONS  SYSTEM  ACQUISITION  REFORM  ACT 

In  2009,  the  United  States  Congress  passed  the  WSARA  into  law  in  an  effort  to 
provide  better  oversight  and  direction  in  the  acquisition  of  defense  programs.  This  law  is 
intended  “to  improve  the  organization  and  procedures  of  the  Department  of  Defense  for 
the  acquisition  of  major  weapon  systems”  (Weapons  System  Acquisition  Reform  Act 
[WSARA],  2009,  Introduction).  With  respect  to  cost  estimation,  the  new  law  imposes 
two  important  requirements: 

1.  The  Director  of  the  Cost  Assessment  and  Program  Evaluation  (CAPE)  shall  “issue 
guidance  relating  to  the  proper  selection  of  confidence  levels  in  cost  estimates 
generally,  and  specifically,  for  the  proper  selection  of  confidence  levels  in  cost 
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estimates  for  major  defense  acquisition  programs  and  major  automated  information 
system  programs”  (WSARA,  2009,  §  101,  subsection  2334(a)). 

2.  The  director  shall  also  “disclose  in  accordance  with  paragraph  (2)  the  confidence 
level  used  in  establishing  a  cost  estimate  for  a  major  defense  acquisition  program  or 
major  automated  information  system  program,  the  rationale  for  selecting  such 
confidence  level,  and,  if  such  confidence  level  is  less  than  80  percent,  the  justification 
for  selecting  a  confidence  level  of  less  than  80  percent”  (WSARA,  2009,  §  101, 
subsection  2334(d)). 

These  two  requirements  drive  the  need  to  alter  the  SBM  of  cost  estimation  into  a 
more  statistically  driven  method.  I  discuss  the  basic  fonn  of  the  SBM  and  the 
enhancements  needed  to  comply  with  the  WSARA  in  Sections  C  and  D  of  this  chapter. 

1.  Non-Statistical  SBM 

The  non-statistical  SBM  method  analyzes  the  risks  of  various  aspects  of  a  project 
in  a  realistic  manner  in  the  opinion  of  the  program  manager  and  the  experts  conducting 
the  analysis.  This  method  brings  to  light  all  possible  and  reasonable  risks  for  a  program 
and  discards  the  extreme  outlier  scenarios.  Figure  1  illustrates  the  non-statistical  SBM, 
and  in  the  subsequent  paragraphs  I  fully  explain  each  step  in  in  the  method. 


Non-statistical  SBM 


PS 


Figure  1.  A  Non-Statistical  SBM  (From  Flynn  and  Garvey,  2011,  p.7) 


The  first  step  in  this  method  is  to  add  up  estimates  of  all  the  elements  from  the 
work  breakdown  structure  (WBS)  as  the  program  is  defined  in  the  program  manager  cost 
analysis  requirements  description  (CARD).  This  sum  of  all  the  elements  of  the  WBS  is 
not  adjusted  for  risk  or  uncertainty.  The  cost  estimated  from  this  step  is  called  the 
program’s  point  estimate  (PE). 
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The  second  step  is  to  define  the  program’s  protect  scenario  (PS).  In  this  step, 
managers  and  analysts  define  the  sources  of  major  known  areas  of  risk  or  uncertainty  that 
have  the  potential  to  impact  the  cost  estimate.  This  PS  should  not  include  worst-case  or 
unrealistic  scenarios  but  should  include  the  areas  in  which  managers  should  focus  their 
attention.  Once  the  PS  is  accepted,  the  cost  associated  with  the  scenario  is  estimated  to 
determine  the  program’s  PS  cost. 

The  difference  between  the  PE  and  the  PS  costs  is  defined  as  the  program’s  cost 
reserve  (CR),  as  illustrated  in  Equation  1.  By  increasing  the  program’s  PS  cost,  the 
amount  of  CR  also  increases.  Conversely,  if  the  PE  increases,  the  CR  will  be  reduced  by 
the  same  amount. 

CR  =  PS  Cost-  P£  (1) 

Analysts  are  then  able  to  conduct  a  sensitivity  analysis  of  the  PS  cost  and  CR  in 
order  to  improve  the  model.  Once  the  model  has  been  completed,  the  program  office  and 
cost  estimators  can  finalize  all  estimates  and  generate  all  of  the  required  reports. 

2.  Statistical  SBM 

The  statistical  SBM  expands  the  non-statistical  method  by  using  the  original  PE 
and  the  addition  of  two  inputs.  These  inputs  are  that  the  probability  PE  will  not  be 
exceeded  ’(o!_PR  ),  and  the  coefficient  of  variation  of  program  cost.  By  using  the 
outputs  from  the  non-statistical  SBM,  basic  algebra,  and  simple  statistical  analysis,  the 
basic  SBM  can  be  transformed  to  produce  a  cumulative  distribution  curve  similar  to  the 
traditional  s-curve  methods. 

The  analysts  provide  the  probability  that  the  PE  cost  will  not  be  exceeded 
H(«;_pe  )  and  the  coefficient  of  variation  (CV)  of  program  cost.  The  probability  that  the 
PE  will  not  be  achieved  is  based  on  the  historical  data  and  judgment  of  the  PM  and  the 
cost  estimators.  The  CV  is  based  solely  on  the  historical  data  from  previous  programs. 
Figure  2  shows  the  steps  of  the  statistical  SMB  process. 
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Inputs  for  Statistical  SBM 

•  Point  estimate  (-Pf  ) 

•  Probability  PE  will  not  be  exceeded  LI®— PE  ) 

•  Coefficient  of  variation  (CV) 


Statistical  SBM 


These  steps  are  the  same  as  the  non-statistical  SBM  process 


Historical  Data 
Guidelines 


These  steps  are  specific  to  the  statistical  SBM  process 
Inputs  aPE  and  the  coefficient  of  variation  (CV)  are  specific  to  the  statistical  SBM  process 


Figure  2.  Statistical  SBM  (From  Flynn  and  Garvey,  201 1,  p. 9) 


The  CV  of  a  probability  distribution  is  the  ratio  of  the  distribution’s  standard 
deviation  to  its  mean.  Equation  2  illustrates  the  calculation  for  determining  a  CV.  The 
CV  lets  cost  estimators  and  analysts  produce  the  probability  distribution  function  (PDF) 
and  cumulative  distribution  function  (CDF). 

CV  =  D  =  - 

»  (2) 

There  are  two  variations  to  the  statistical  SBM  to  account  for  the  difference 
between  normal  and  lognormal  probability  distributions  (Flynn  and  Garvey,  2011,  p.10- 
11).  Equations  3-6  illustrate  how  the  statistical  SBM  inputs  are  used  to  calculate  the 
program’s  mean  cost  and  cost  standard  deviation. 

Normal  Probability  Equations 

D  Xjjj  J 

Moon^  -  *«  "  z«i+0.;Ire)  (3) 
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"  1  +  Diln) 


D^pg,.' 


(4) 


Lognormal  Probability  Equations 


1 


Wbbokwb"  *»«„-  2?z^ln*'l  -b  D2) 


(5) 


<hnCertBpn  -  v’lnvl  -b  D\) 


(6) 


The  lognormal  mean  and  standard  deviation  require  conversion  from  log-dollars 
to  dollars  by  Equations  7  and  8  (Garvey,  2005). 


(7) 


Analysts  are  now  able  to  conduct  a  sensitivity  analysis  of  the  model  by  altering 
the  «pe  and  the  CV.  Alterations  can  create  steeper  or  shallower  slopes  to  the  CDF,  as 
well  as  shift  the  mean  of  the  function.  These  alterations  to  the  SBM  allow  cost  estimators 
to  produce  reports  that  are  similar  to  the  widely  accepted  s-curve  outputs. 

C.  NCCA  STUDY  AND  THE  ENHANCED  SBM 

In  Section  C,  I  demonstrated  the  methods  necessary  to  produce  confidence 
intervals  and  CDFs  for  the  SBM.  The  alteration  to  the  and  the  CV  allows  cost 
estimators  to  produce  reports  that  are  similar  to  the  widely  accepted  s-curve  outputs.  The 
issue  of  how  to  ground  the  SBM  in  historical  data  still  remains  problematic. 

In  201 1,  Flynn  and  Garvey  conducted  research  of  historical  costs  across  all  realms 
of  DoD  acquisitions  in  order  to  integrate  historical  data  into  the  SBM.  The  research 
focused  on  detennining  the  appropriate  value  for  the  program’s  and  CV. 

The  Flynn  and  Garvey  study  derived  a  value  of  «?£  -0.34  for  programs  at 
Milestone  B  based  on  a  historical  CV  equal  to  0.51  (Flynn  &  Garvey,  2011).  Similar 
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findings  were  noted  in  the  2007  RAND  report,  which  noted  a  =  0.09  based  on  an 
implied  CV  =  0.26  at  Milestone  B  (Younossi,  2007).  These  two  reports  give  analysts  a 
bounded  region  of  «pe  to  work  within.  Now  that  a  historical  value  range  for  the  «pi  has 
been  derived,  the  only  input  left  to  consider  is  the  CV. 

In  the  Flynn  and  Garvey  study,  research  focused  on  detennining  what  actually 
drives  the  value  of  the  CV  for  various  programs.  Their  first  conjecture  was  to  determine 
whether  cost  growth  factors  (CGFs)  have  been  historically  consistent.  The  researchers 
found  routine  underestimation  of  cost  within  the  DoD.  The  second  conjecture  was  that 
CVs  decreased  throughout  the  acquisition  process.  The  researchers  confirmed  this 
assumption  of  decreasing  CVs  as  programs  progressed  through  the  acquisition  life  cycle. 
The  third  conjecture  was  that  CVs  were  equivalent  for  all  acquisition  programs  regardless 
of  platform.  Again,  the  researchers  proved  this  assumption  to  be  true,  especially  from 
early  milestones,  such  as  Milestone  B.  The  forth  conjecture  was  that  CVs  would  be 
lower  if  they  were  adjusted  to  account  for  quantity  changes  and  actual  inflation.  Again, 
the  NCCA  researchers  supported  this  assumption  based  on  historical  data.  The  fifth 
conjecture  was  that  duration  would  have  no  effect  on  the  CV.  The  researchers  found  no 
evidence  to  support  this  assumption;  the  data  actually  suggests  quite  the  opposite  (Flynn 
&  Garvey,  2011). 

D.  SUMMARY 

The  possibility  of  producing  historically  based  cost  estimates  with  minimal 
statistical  analysis  has  driven  the  desire  for  a  new  method  of  cost  estimation.  For  any 
new  method  to  be  useful  in  the  cost-estimating  community,  it  would,  as  a  minimum,  have 
to  meet  the  requirements  of  the  WSARA.  The  non-statistical  SBM  produces  a  cost 
estimation  method  that  is  bounded  by  a  scenario  agreed  upon  by  the  program 
management  team;  it  also  derives  the  program’s  PE.  The  program  management  team’s 
PE  is  coupled  to  a  probability  of  achieving  the  estimated  PE  and  a  coefficient  of  variation 
to  allow  for  statistically  adjusted  cost  estimates. 
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During  Flynn  and  Garvey’s  study  of  program  cost  estimates,  the  eSBM  gained  the 
historical  perspective  necessary  to  achieve  credibility  with  other  cost  estimators.  The 
evolution  of  the  SBM  into  the  eSBM  has  revealed  an  all-new  possibility  of  risk-adjusted 
cost  estimation. 
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III.  METHODS  OF  ANALYSIS 


A.  NCCA  HISTORICAL  DATABASE 

1.  How  the  Database  Was  Created 

The  cornerstone  of  any  research  project  is  the  collectable  data.  The  NCCA 
collected  and  organized  raw  data  from  100  Selected  Acquisitions  Reports  (SARs).  The 
SARs  were  mostly  historical  DoN  major  acquisition  programs  that  were  at  various  stages 
of  completion  (Flynn  &  Garvey,  2011).  The  database  organized  cost  data  in  current-year 
and  then-year  values,  quantity  adjustments,  information  on  platform  type,  milestone  cost 
estimates,  and  annual  program  cost  updates.  By  creating  a  well-functioning  database,  the 
groundwork  for  further  analysis  had  been  established. 

2.  Assumptions  Used  in  the  Database 

The  SAR  is  the  best  source  of  data  for  a  program  from  a  managerial  standpoint. 
The  two  key  cost  estimates  present  in  the  SAR  are  the  Baseline  Estimate  (BE)  and  the 
Current  Estimate  (CE).  Although  the  cost  estimates  in  the  SAR  are  not  perfect,  numerous 
cost  studies  have  indicated  that  the  SAR  is  the  best  source  for  comprehensive  program 
data  (Flynn  &  Garvey,  2011,  p.21).  For  the  NCCA  research  as  well  as  follow-on 
research,  the  BE  is  used  as  the  initial  cost  estimate,  and  the  CE  of  the  last  SAR  for  a 
program  is  used  as  the  final  actual  cost.  As  stated  by  Flynn  and  Garvey  (2011)  in  the 
eSBM, 

based  on  an  analysis  of  10  programs  in  our  database  dating  from  the 

1990s,  there  is  little  difference  between  the  SAR  BE,  the  program  office 

estimate,  and  the  Independent  Cost  Estimate  (ICE)  conducted  either  by  the 

Naval  Center  for  Cost  Analysis  (NCCA)  or  the  Office  of  the  Secretary  of 

Defense  (OSD),  (p.  21) 

B.  NCCA  ANALYSIS  METHOD 

In  order  to  detennine  the  CGF,  the  NCCA  study  used  the  ratio  of  BE  to  CE.  As 
stated  in  Chapter  II  Equation  2,  the  CV  is  the  ratio  between  the  sample  standard  deviation 


13 


and  the  sample  mean.  For  the  different  hypotheses  in  the  NCCA  study,  the  mean  and 
standard  deviation  for  each  set  of  CGF  was  calculated  to  determine  the  applicable  CV. 

1.  Acquisition  Milestone  Adjustment 

In  the  NCCA  study,  researchers  calculated  the  CGF  at  Milestones  A,  B,  and  C  for 
all  programs  in  the  database  regardless  of  platfonn  or  type.  They  then  plotted  the  results, 
which  revealed  a  lognormal  distribution  of  CGFs  for  both  the  Milestones  B  and  C.  There 
was  insufficient  data  to  determine  the  distribution  for  data  at  Milestone  A.  The 
researchers  used  statistical  analogy  to  detennine  the  approximate  CVs  for  Milestone  A. 
After  calculating  the  CGFs  for  each  program,  researchers  then  calculated  the  CV  for  each 
program  using  both  quantity  and  non-quantity  adjustments  for  base-  and  then-year 
dollars.  They  then  plotted  the  mean  and  range  for  CVs  at  Milestones  A,  B,  and  C  to 
reveal  lower  CVs  for  the  later  milestones.  The  more  accurate  cost  estimation  can  be 
explained  by  technology  maturation  as  well  as  by  clearer  information  about  program 
parameters.  It  is  noted  in  the  NCCA  study  that  the  range  of  the  CVs  narrowed  at  the  later 
milestones.  Figure  3  is  an  illustration  of  the  results  of  this  analysis. 


Figure  3.  Benchmark  CVs  by  Acquisition  Milestone — Quantity  Adjusted  in  Then- 

Year  Dollars  (From  Flynn  &  Garvey,  2011,  p.28) 

2.  Platform  Homogeneity 

In  the  NCCA  study,  researchers  also  analyzed  the  difference  in  cost  growth 

between  platforms.  For  this  part  of  the  study,  researchers  separated  the  CGFs  by  program 

type.  The  type  categories  included  ships  and  subs,  aircrafts,  missiles,  electronics,  and 

other.  Researchers  calculated  the  mean  and  variance  for  each  category  to  allow  for  an 
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analysis  across  categories.  The  surprising  finding  of  this  portion  of  the  study  was  that  the 
CGFs  across  platforms  and  categories  were  consistent.  At  both  Milestones  B  and  C,  the 
CGFs  were  within  the  5%  alpha  confidence  level.  This  finding  proved  that  there  is 
significant  homogeneity  between  programs.  Figures  4  and  5  illustrate  the  relationship  of 
CGF  to  platform  type  at  Milestones  B  and  C. 
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Figure  4.  Milestone  B:  CGF  Means  and  Variances  (Quantity  Adjusted  in  Then- Year 

Dollars)  (From  Flynn  &  Garvey,  2011,  p.24) 
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Figure  5.  Milestone  C:  CGF  Means  and  Variances  (Quantity  Adjusted  in  Then- Year 

Dollars)  (From  Flynn  &  Garvey,  2011,  p.27) 


3.  Quantity  Normalization 

To  account  for  changes  in  the  quantity  of  items  being  procured,  researchers  in  the 
NCCA  study  analyzed  the  CV  first  without  adjustment  for  changes  in  quantity  and  then 
adjusted  for  changes  in  quantity.  It  was  no  surprise  that  the  adjusted  CVs  were  lower 
than  the  unadjusted  CVs.  It  is  the  opinion  of  the  NCCA  researchers  that  “changes  in 
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acquisition  quantity  from  a  program  baseline  are  generally  regarded  as  beyond  the 
purview  of  the  cost  analyst”  (Flynn  &  Garvey,  201 1,  p.  22).  The  researchers  conducted 
the  quantity  adjustment  in  two  manners  to  determine  whether  the  adjustment  method  had 
any  effect  on  the  calculated  CVs.  First,  researchers  scaled  the  BE  to  account  for  the 
actual  number  produced  or  acquired;  next,  they  nonnalized  the  CE  to  reflect  the  actual 
cost  if  the  original  quantity  was  delivered.  The  differences  between  the  two  methods 
proved  to  be  insignificant;  however,  the  differences  between  the  quantity-adjusted  and 
the  non-quantity-adjusted  CVs  were  significant. 

4.  Inflation  Normalization 

Although  SARs  are  adjusted  to  inflation,  the  values  used  are  predicted  inflation 
rates,  not  the  actual  rates  observed  during  the  time  period  covered  by  the  program’s  life 
cycle.  Inflation  rates  are  promulgated  by  the  OSD  to  ensure  that  there  is  a  standard  set  of 
inflation  values  used  across  all  areas  of  the  DoD.  The  problem  arises  when  the  actual 
rates  are  drastically  different  than  the  predicted  rates  from  OSD.  An  example  of  drastic 
differences  between  the  two  rates  occurred  during  the  Carter  administration  when  the 
observed  rate  was  nearly  10.7%,  whereas  the  OSD  was  forecasting  rates  between  3%  and 
4%  (Flynn  &  Garvey,  2011).  By  the  time  the  OSD  began  to  change  the  rates,  inflation 
was  on  the  decline  and  the  OSD  rates  over  estimated  inflation  during  the  Reagan 
administration  (Flynn  &  Garvey,  2011).  The  researchers  of  the  NCCA  study  calculated 
CVs  using  the  original  inflation  rates  and  also  using  the  actual  historical  inflation  rates. 
Once  again,  they  confirmed  the  expected  effect  of  lower  CVs  for  the  historical  inflation- 
adjusted  estimates. 

C.  METHOD  FOR  ACQUISITION  STRATEGY  CV  ANALYSIS 

For  my  analysis  of  the  effect  of  acquisition  strategy  on  the  CGF  and  CV,  I  placed 
each  SAR  in  a  category  corresponding  to  the  type  of  acquisition  strategy  used  to  procure 
the  item.  Once  the  programs  were  separated  into  their  respective  acquisition  strategies,  I 
computed,  for  each  category,  its  standard  deviation  and  mean.  From  the  mean  and 
standard  deviation,  I  was  able  to  calculate  the  CV  for  each  acquisition  strategy.  To 
ensure  that  the  changes  in  the  CGF  and  CV  were  not  due  to  factors  other  than  the 
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acquisition  strategy,  I  adjusted  the  values  for  the  BE  and  CE  to  account  for  changes  in 
quantity  and  for  historical  inflation  rates. 


D.  METHOD  FOR  PROGRAM-LENGTH  C V  ANALYSIS 

To  analyze  the  effect  of  program  length  on  the  CV,  I  measured  each  program’s 
total  length  and  added  the  data  to  the  NCCA  database.  I  calculated  the  CGF  for  each 
program  and  then  plotted  the  CGF  against  the  program  life  span.  Once  the  CGFs  were 
plotted  with  respect  to  time,  I  detennined  the  mean  and  standard  deviation  for  the  entire 
sample  set  so  that  I  could  derive  an  overall  CV  for  program  length. 

I  also  segregated  the  CGF  into  three  general  time  frames.  The  time  frames  were 
short  acquisition  lengths  (<  5  years),  average  acquisition  lengths  (sample  average  +/-  one 
standard  deviation),  and  long  acquisition  lengths  (>  one  standard  deviation  longer  than 
the  average).  For  each  category,  I  then  had  a  CV  calculated  to  determine  a  relationship 
between  the  program  length  and  the  expected  CV.  I  adjusted  the  values  for  the  BE  and 
CE  to  account  for  changes  in  quantity  and  for  historical  inflation  rates  to  ensure  that  the 
effects  of  the  variation  were  due  to  program  length  and  not  inflation  or  quantity  changes. 

E.  SUMMARY 

The  NCCA  database  of  over  100  SARs,  spanning  the  last  4  decades,  is  the  single 
best  resource  of  data  on  historical  cost.  The  key  assumption  used  in  the  NCCA  study  was 
that  the  SAR  BE  would  be  used  as  the  program’s  initial  cost  estimate,  and  that  the  CE 
would  be  used  as  the  program’s  final  cost.  To  normalize  the  initial  estimates,  the  BE 
would  have  to  be  adjusted  to  account  for  changes  in  quantity  and  changes  in  actual 
inflation  from  the  estimated  inflation  rates  used  in  the  cost-estimation  process.  A  ratio  of 
the  BE  and  the  CE  are  used  to  calculate  the  program’s  CGF.  The  NCCA  researchers 
calculated  the  CV  for  various  combinations  of  CGFs  to  detennine  the  appropriate  CV  to 
use  in  the  eSBM.  Additional  research  still  needs  to  be  conducted  regarding  acquisition 
strategy  and  overall  program  length.  Comparing  the  CGFs  for  the  different  acquisition 
strategies  and  program  lengths  will  provide  further  insight  into  the  drivers  for  program 
CVs. 
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IV.  ANALYSIS 


A.  INTRODUCTION 

In  this  section,  I  describe  the  findings  for  the  analysis  of  program  length  and 
acquisition  strategy.  Section  B,  in  which  I  discuss  program  length,  is  divided  into  three 
parts:  all  programs,  completed  programs,  and  on-going  programs.  Section  C,  in  which  I 
discuss  acquisition  strategy,  illustrates  the  problems  with  correlating  any  one  acquisition 
strategy  with  a  CGF. 

B.  PROGRAM  LENGTH 

I  conducted  the  data  analysis  for  program  length  to  determine  a  correlation 
between  program  length  and  CGFs.  I  grouped  the  programs  into  two  different  categories: 
completed  and  on-going.  Additionally,  I  developed  a  third  group  of  programs  to  include 
all  programs.  In  the  following  sections,  I  discuss  the  findings  for  the  claim  that  the 
longer  the  programs  length,  the  more  cost  growth  can  be  expected. 

1.  All  Programs 

I  used  the  all  programs  group  as  the  control  to  determine  any  difference  between 
the  findings  of  Flynn  and  Garvey  and  my  follow-on  research  (2011).  The  sample  data 
that  I  used  included  50  Milestone  B  cost  estimates.  I  redid  the  analysis  for  CGFs  and 
CVs  for  all  programs  to  verily  Flynn  and  Garvey’s  results.  My  results  were  the  exact 
same  as  those  illustrated  in  Flynn  and  Garvey’s  2011  paper  on  the  eSBM.  Their  findings 
make  a  clear  connection  between  the  CGF  and  the  quantity  adjustment.  Table  1 
illustrates  the  Milestone  B  CGFs  and  CVs  for  all  programs.  Additionally,  the 
relationship  between  base-year  and  then-year  adjustments  illustrates  the  relationship 
between  the  CGF  and  inflation  for  the  same  programs. 
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Cost  Growth  Factors  &  CVs  for  All  DON  MDAPs  at  MS  B;  n  =  49 


(Without  Qty  Adjustment) 

(Quantity  Adjusted) 

Statistics 

Base-Year$ 

Then-Year$ 

Base-Year$ 

Then-Year$ 

Mean 

1.49 

1.86 

1.24 

1.36 

Standard 

Deviation 

0.94 

1.61 

0.44 

0.69 

C  V 

0.63 

0.87 

0.36 

0.51 

Table  1 .  Cost  Growth  Factors  and  CVs  at  Milestone  B  (All  Programs) 

I  analyzed  the  relationship  between  the  CGF  and  program  length  for  all  programs. 
I  assumed  that  no  further  schedule  or  performance  growth  would  occur  for  the  programs 
that  were  still  on  going.  I  conducted  a  regression  analysis  on  all  50  programs.  The  most 
effective  regression  was  the  linear  regression  for  all  four  cases:  non-quantity-adjusted 
base-year  dollars  and  then-year  dollars,  and  quantity-adjusted  base-year  dollars  and  then- 
year  dollars.  Figures  6  and  7  illustrate  the  linear  regression  for  all  four  cases.  Figure  6 
shows  a  slight  correlation  between  CGFs  and  program  length  for  the  non-quantity- 
adjusted  cases.  The  correlation  is  not  as  evident  in  Figure  7  for  the  quantity-adjusted 
case. 

The  regression  results  for  all  four  cases  yielded  insignificant  results  in 
determining  a  correlation  between  the  CGFs  and  the  programs  length.  The  results  were 
insignificant  at  the  90%  level,  thus  leading  me  to  reject  the  hypothesis  that  there  is 
correlation  between  the  two  variables.  That  is,  I  concluded  that  there  is  no  correlation 
between  the  two  variables.  The  analysis  also  failed  the  F  test  and  t  test,  confirming  the 
lack  of  correlation.  The  /^-squared  and  adjusted-fCsquared  results  for  the  non-quantity 
regression  had  a  minimal  significance;  however,  once  quantity  adjustments  were  made, 
there  was  no  longer  any  noticeable  significance.  These  results  deny  the  claim  that 
program  length  has  any  effect  on  program  cost  growth. 
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Figure  6.  All  Programs  Non-Quantity-Adjusted  Linear  Regression 
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Figure  7.  All  Programs  Quantity- Adjusted  Linear  Regression 
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2.  Completed  Programs 

For  the  completed  programs,  the  only  assumption  I  made  was  that  there  were  no 
outstanding  costs  for  the  completed  programs.  This  is  a  fair  assumption  considering  the 
amount  of  time  that  had  elapsed  for  a  majority  of  the  completed  programs.  My  first  task 
was  to  conduct  regression  analysis  of  the  CGFs  for  all  completed  programs,  from  the 
Milestone  B  estimates  against  the  program  length.  I  set  up  the  analysis  to  conduct  all  the 
nonnal  regressions  to  include  linear,  exponential,  and  logarithmic  regressions  to  produce 
the  most  significant  regression  result.  The  linear  regression  produced  the  most 
significant  results  for  both  quantity-  and  non-quantity-adjusted  values  of  base-year 
dollars  and  then-year  dollars.  Figures  8  and  9  illustrate  the  linear  regressions  between  the 
program  length  on  the  x  axis  and  the  CGF  on  the  y  axis.  Figure  8  suggests  only  minor 
significance  to  the  claim  that  program  length  has  some  effect  on  the  CGF;  however,  once 
the  program  was  adjusted  for  quantity  changes,  the  value  for  /^-squared  was  reduced  to  an 
insignificant  value,  as  illustrated  in  Figure  9.  The  adjusted  /^-squared  values  revealed  an 
even  lower  significance  of  the  relation  between  the  CGF  and  the  program  length  to  claim 
that  program  length  has  an  effect  on  program  CGFs  for  completed  programs.  The  non¬ 
quantity-adjusted  values  produced  results  with  modest  correlation  but  that  were  still  not 
above  the  90%  confidence  level.  Once  I  ran  the  correlations  for  quantity-adjusted  CGFs, 
the  correlation  became  even  worse:  all  four  analyses  failed  both  the  F  test  and  t  test  for 
significance.  I  rejected  the  claim  that  program  length  affects  cost  growth  for  completed 
programs  at  the  Milestone  B  cost  estimate. 
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Figure  8.  Completed  Programs  Non-Quantity- Adjusted  Linear  Regression 
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The  next  analysis  I  conducted  was  to  detennine  the  average  CGF  and  CV  for  the 
completed  programs.  Table  2  illustrates  the  CGFs  and  CVs  for  the  completed  programs 
from  the  Milestone  B  estimates.  These  values  are  consistent  with  Flynn  and  Garvey’s 
findings  illustrated  in  Table  1  (2011).  These  findings  continue  to  validate  the  claim  that 
changes  in  program  quantity  play  the  most  significant  role  in  determining  cost  growth. 
The  results  of  the  completed  programs  produced  more  significant  results  than  the  results 
of  all  programs  due  to  the  removal  of  the  on-going  programs. 


Cost  Growth  Factors  &  CVs  for  Completed  DON  MDAPs  at  MS  B;  n  =  35 


(Without  Qty  Adjustment) 

(Quantity  Adjusted) 

Statistics 

Base-Year$ 

Then-Year$ 

E3ase-Year$ 

Then-Year$ 

Mean 

1.61 

2.10 

1.26 

1.40 

Standard 

Deviation 

1.07 

1.84 

0.48 

0.78 

CV 

0.66 

0.87 

0.38 

0.56 

Table  2.  Cost  Growth  Factors  and  CVs  at  Milestone  B  (Completed  Programs) 

3.  On-Going  Programs 

The  results  for  on-going  programs  revealed  a  negative  relationship  between  CGFs 
and  program  length.  The  negative  cost  growth  for  programs  still  on  going  is  not  expected 
and  counter-intuitive  to  the  results  of  completed  programs.  Programs  with  the  most  time 
remaining  until  completion  have  the  lower  CGFs.  Also,  the  differences  between  non¬ 
quantity-adjusted  and  quantity-adjusted  CGFs  are  not  as  obvious,  since  many  of  the 
programs  have  yet  to  be  affected  by  quantity  drawdowns.  Additionally,  many  of  the 
programs  have  not  been  in  existence  long  enough  for  the  effects  of  inflation  to  become 
relevant.  Figures  10  and  11  show  the  negative  relationship  between  the  CGFs  and  the 
program  length  for  on-going  programs.  The  regressions  for  the  on-going  programs  also 
show  large  standard  errors  and  low  adjusted  R- squared  values. 
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There  is  some  significance  at  the  90%  confidence  level  for  all  four  cases;  however,  there 
are  only  10  observations  that  reduce  the  reliability  of  these  findings.  Once  again,  the 
findings  of  this  research  contradict  the  claim  that  program  length  has  any  effect  on  cost 
growth. 


BY$  and  TY$  w/out  Quantity  Adjustment 
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Figure  10.  On-Going  Programs  Non-Quantity- Adjusted  Linear  Regression 
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Table  3  shows  the  average  CGFs  and  CVs  for  the  on-going  programs.  Even  for 
the  on-going  programs,  the  CGFs  and  CVs  for  the  quantity-adjusted  values  are  within  the 
range  of  the  completed  program  threshold. 
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Cost  Growth  Factors  &  CVs  for  On-Going  DON  MDAPs  at  MS  B;  n  =  10 


(Without  Qty  Adjustment) 

(Quantity  Adjusted) 

Statistics 

Base-Year$ 

Then-Year$ 

Base-Year$ 

Then-Year$ 

Mean 

1.25 

1.33 

1.27 

1.35 

Standard 

Deviation 

0.41 

0.46 

0.37 

0.46 

CV 

0.32 

0.35 

0.29 

0.34 

Table  3.  Cost  Growth  Factors  and  CVs  at  Milestone  B  (On-Going  Programs) 

C.  ACQUISITION  STRATEGY 

1.  Types  of  Acquisition  Strategies 

The  analysis  of  acquisition  strategy  resulted  in  more  questions  and  no  definitive 
answers.  The  problem  with  relating  any  one  acquisition  strategy  to  a  CGF  is  the 
complexity  of  the  acquisition  system  and  modern  program  management.  Program 
managers  use  a  mixture  of  contract  types  during  the  Milestone  B  and  beyond  the 
contracts  negotiation  stage.  For  the  current  on-going  programs,  each  program  uses  a 
mixture  of  firm-fixed-price  (FFP),  fixed-price-incentive-fee  (FPIF),  cost-plus-fixed-fee 
(CPFF),  cost-plus-incentive-fee  (CPIF),  and  cost-plus-award-fee  (CPAF)  contract  types. 
The  problem  of  assigning  acquisition  strategy  to  a  single  contract  type  becomes  more 
difficult  when  the  issue  of  awards  and  incentive  fees  is  taken  into  account.  Awards  and 
incentives  are  written  into  contracts  at  the  beginning  but  not  awarded  until  contract 
completion. 

2.  Continuation  of  Research 

The  next  step  in  trying  to  determine  how  acquisition  strategy  affects  cost  growth 
is  to  break  down  cost  growth  for  each  contract  type  into  individual  CGFs.  Once  the 
individual  CGFs  have  been  determined,  they  can  be  weighted  to  account  for  the  relative 
amount  each  contract  contributes  to  the  total  program  cost.  Breaking  down  the  programs 
to  the  next  lowest  level  will  allow  for  further  insight  into  the  drivers  for  cost  growth. 
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D.  SUMMARY 

To  the  95th  percentile  confidence  level,  I  have  rejected  the  hypothesis  that  as 
program  length  increases  so  does  the  CGF.  Furthermore,  the  findings  of  this  research 
have  confirmed  the  findings  of  the  NCCA  study  on  CGFs  and  historical  CVs.  The 
analysis  of  program  length  yielded  no  significant  relationship  between  the  program 
length  and  CGFs.  The  analysis  of  program  length  reiterated  the  effects  of  quantity 
changes  on  CGFs  as  a  significant  driver  for  cost  growth.  Additionally,  the  effects  of 
inflation  on  program  cost  growth  continue  to  plague  the  cost-estimation  community. 
Neither  the  changes  in  quantity  nor  the  changes  in  inflation  rates  are  under  the  control  of 
the  cost  estimators;  however,  these  changes  must  be  accounted  for  in  the  cost  risk  profile. 
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V.  CONCLUSION 


A.  SUMMARY 

In  Chapter  I,  I  discussed  the  basis  for  this  research  and  the  issue  of  risk-adjusted 
cost  estimation  using  the  Monte  Carlo  simulation  method.  The  defining  issues  with  cost 
estimation  at  the  Milestone  B  include  the  uncertainty  of  a  clear  program  direction,  the 
difficult  statistical  processes,  and  the  unbounded  reality  of  possible  consequences.  These 
difficulties  lead  to  the  need  for  a  new  method  of  calculating  risk-adjusted  cost  estimation. 
A  new  method  would  be  beneficial  to  everyone  involved  in  the  acquisition  process,  from 
the  MDA  to  the  cost  estimators. 

In  Chapter  II,  Literature  Review,  I  described  Paul  Garvey  and  Brian  Flynn’s 
findings  as  a  more  effective  method  of  risk-adjusted  cost  estimation  (2011).  This  chapter 
started  with  the  necessary  requirements  defined  in  the  WSARA.  The  WSARA  tasks  the 
Director  of  CAPE  with  issuing  guidance  on  the  appropriate  confidence  intervals  to  be 
used  and  with  justifying  their  use  if  that  level  is  below  the  80%  threshold.  Garvey’s 
SBM  provided  the  framework  for  a  simplified  method  of  risk-adjusted  cost  estimation 
using  the  statistical  SBM  (2005).  Flynn  and  Garvey’s  study  on  CGFs  and  CVs  led  to 
enhancements  in  the  SBM  that  aligned  the  new  eSBM  with  the  requirements  of  the 
WSARA  (2011). 

Chapter  III,  Methods  of  Analysis,  includes  two  parts:  what  has  been  done  in  the 
past,  and  what  this  thesis  does  differently.  The  NCCA  study  brought  to  light  the  issue  of 
inaccurate  inflation  estimates  and  quantity  adjustments  as  a  significant  reason  for  cost 
growth.  The  NCCA  study  also  debunked  the  theory  that  there  are  secular  differences  in 
cost  growth  across  program  types.  The  methods  used  in  this  thesis  were  aimed  at  relating 
cost  growth  to  program  length  and  acquisition  strategy. 

I  based  my  analysis  in  this  thesis  on  two  hypotheses:  first,  that  cost  growth  will 
increase  as  the  programs’  lengths  also  increase;  and  second,  that  different  acquisitions 
strategies  will  yield  different  rates  of  cost  growth.  I  rejected  the  first  hypothesis  with  a 
high  degree  of  confidence  because  I  noted  no  correlation  between  the  program  length  and 


29 


CGF.  My  analysis  of  program  length  did,  however,  further  support  Flynn  and  Garvey’s 
findings  that  inflation  and  quantity  adjustment  are  the  most  significant  sources  of  cost 
growth  (2011).  During  my  analysis  of  the  second  hypothesis,  I  discovered  that  no  single 
acquisition  strategy  is  used  in  contracting  an  MDAP.  The  complexity  of  acquisition 
strategy  made  the  second  hypothesis  impossible  to  validate  in  the  manner  that  I 
illustrated. 

B.  RECOMMENDATIONS 

I  recommend  two  main  areas  for  continued  research  in  historical  cost  growth  and 
potential  cost  drivers. 

The  first  major  area  that  needs  to  be  researched  is  in  contract  management  and  in 
the  methods  used  as  a  possible  cost  driver.  This  area  should  include  a  breakdown  of  each 
program  to  the  contract  level  to  determine  the  cost  growth  for  different  contact  types,  and 
then  the  contract  types  across  programs  could  be  compared  as  a  potential  link  to  cost 
growth. 

The  second  area  in  which  research  could  be  beneficial  is  the  effect  of  awards  and 
incentives  on  cost  growth.  An  analysis  of  what  percentage  of  cost  growth  is  accounted 
for  in  the  awards  and  incentives  could  lend  insight  into  how  to  more  accurately  estimate 
awards  and  incentives  for  MDAPs. 

Other  areas  where  research  is  still  needed  in  the  risk-adjusted  cost  estimation 
arena  include 

1 .  how  the  eSBM  compares  to  the  Monte  Carlo  simulation  at  later  milestones 

2.  how  to  account  for  and  eliminate  PM  bias  during  the  cost-estimation  process. 
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